We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale, encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at https://ogb.stanford.edu .

Weaknesses: - The contribution goes in several directions which makes the paper hard to evaluate; is the main contribution the selection of existing datasets, introducing new datasets or new versions of datasets, the empirical evaluation or the software tooling? - The dataset does not describe existing datasets and benchmarks, and so it is hard to judge the exact differences between the proposed datasets and currently used datasets. A more direct comparison might be useful, and it's not clear why existing, smaller datasets are not included in the collection. In particular, the datasets from Moleculenet seem to be mostly reproduced, using the splitting strategy that was suggested in their paper, with the modification potentially being addition of new features. What was the pool of datasets that was drawn from, and how were datasets selected? An example of such a work is the OpenML100 and OpenML CC-18 for classification, see Bischl et.

The paper introduces open graph benchmark, a collection of 15 datasets, as well as software for reproducible benchmarking. Three knowledgeable reviewers support acceptance for the contributions of the paper, and I also recommend to accept for NeurIPS 2020. However, I encourage the authors to address the concerns raised by the reviewers in the final version, in particular to reword the claims of the paper as mentioned by Reviewers #4 and #5.

We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale, encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation.